Image processing apparatus, image processing method, and recording medium

ABSTRACT

An image processing apparatus ( 100 ) includes: a variable determination unit ( 134 ) that performs, based on a distribution of luminance values of individual pixels included in a predetermined area determined according to information regarding a motion of a user out of first video data recorded in a first dynamic range, determination of a variable to be used to calculate the luminance values of the individual pixels when the first dynamic range is converted into a second dynamic range; and a converter ( 135 ) that converts the first video data into second video data displayed in the second dynamic range, based on the variable determined by the variable determination unit.

FIELD

The present disclosure relates to an image processing apparatus, animage processing method, and a recording medium. Specifically, thepresent disclosure relates to conversion of the dynamic range of videodata.

BACKGROUND

With the spread of virtual reality (VR) technology, spherical camerascapable of omnidirectional imaging in 360 degrees are widely used.Moreover, devices such as a head-mounted display (HMD) have begun tospread as a viewing environment for spherical content such as sphericalimages and spherical movies captured by a spherical camera.

Here, there is a known technology that dynamically adjusts thebrightness of the image according to the user's line-of-sight directionto artificially increase the dynamic range during imaging in order toimprove the visibility when viewing spherical content on the HMD.

CITATION LIST Patent Literature

Patent Literature 1: JP 2017-22665 A

SUMMARY Technical Problem

However, with the above-described conventional technique, it is notalways possible to achieve high-clarity brightness/darkness expressionof the image. For example, the related art calculates the mean of thebrightness of pixels in a predetermined range of video data and thenadjusts the calculated mean of the brightness to a certain range ofbrightness. However, uniformly adjusting the brightness of the imageonly changes the brightness of the entire image, and this leads to afailure in achieving high-clarity brightness/darkness expression withinthe image. For example, uniformly adjusting the brightness of the imagemight sometimes lead to a failure in preventing crushed shadows orblown-out highlights in a converted image.

Meanwhile, the above-described problem can also occur in situationsother than the situation in which spherical content is viewed on an HMD.For example, when reproduction of the content in the original dynamicrange is difficult in the user's viewing environment, the content isreproduced within a converted dynamic range. In a case where the dynamicrange is converted at the time of reproduction of content from the timeof recording of the content, high-clarity brightness/darkness expressionof the converted image might not be achieved.

In view of this, the present disclosure proposes an image processingapparatus, an image processing method, and a recording medium capable ofachieving a high-clarity brightness/darkness expression of an image.

Solution to Problem

According to the present disclosure, an image processing apparatusincludes: a variable determination unit that performs, based on adistribution of luminance values of individual pixels included in apredetermined area determined according to information regarding amotion of a user out of first video data recorded in a first dynamicrange, determination of a variable to be used to calculate the luminancevalues of the individual pixels when the first dynamic range isconverted into a second dynamic range; and a converter that converts thefirst video data into second video data displayed in the second dynamicrange, based on the variable determined by the variable determinationunit.

(Operation) The image processing apparatus of the present disclosureperforms image processing by selectively using pixels included in apredetermined area determined according to the motion of a user, ratherthan by the pixel information of the first video data as a whole.

Specifically, the image processing apparatus of the present disclosuredynamically determines a variable used at converting video data, basedon the distribution of the luminance values of individual pixelsincluded in a predetermined area. In this manner, the image processingapparatus according to the present disclosure uses the pixel informationselected according to the motion of the user to interactively change theconversion process of the video data according to the motion of theuser.

Advantageous Effects of Invention

With the image processing apparatus, the image processing method, andthe recording medium according to the present disclosure, it is possibleto achieve a high-clarity brightness/darkness expression of an image. Itshould be noted that the effects described herein are not necessarilylimited and may be any of the effects described in the presentdisclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of an image processingsystem according to a first embodiment of the present disclosure.

FIG. 2 is a histogram illustrating a distribution of luminance values ofpixels included in an image.

FIG. 3 is a diagram illustrating an example of image processingaccording to the first embodiment of the present disclosure.

FIG. 4 is a diagram (1) illustrating an example of image processingaccording to the first embodiment of the present disclosure.

FIG. 5 is a diagram (2) illustrating an example of image processingaccording to the first embodiment of the present disclosure.

FIG. 6 is a flowchart illustrating an example of a flow of a processaccording to the first embodiment of the present disclosure.

FIG. 7 is a diagram illustrating an example of image processingaccording to a modification of the first embodiment of the presentdisclosure.

FIG. 8 is a diagram illustrating an example of an image processingsystem according to a second embodiment of the present disclosure.

FIG. 9 is a diagram illustrating an example of image processingaccording to the second embodiment of the present disclosure.

FIG. 10 is a diagram illustrating an example of image processingaccording to the second embodiment of the present disclosure.

FIG. 11 is a flowchart illustrating a flow of a process according to thesecond embodiment of the present disclosure.

FIG. 12 is a diagram illustrating an example of an image processingsystem according to a third embodiment of the present disclosure.

FIG. 13 is a hardware configuration diagram illustrating an example of acomputer that realizes functions of an image processing apparatus.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described below in detailwith reference to the drawings. In each of the following embodiments,the same parts are denoted by the same reference symbols, and arepetitive description thereof will be omitted.

The present disclosure will be described in the following item order.

1. First embodiment

1-1. Configuration of image processing system according to firstembodiment

1-2. Example of image processing according to first embodiment

1-3. Image processing procedure according to first embodiment

2. Modification of first embodiment

3. Second embodiment

3-1. Configuration of image processing system according to secondembodiment

3-2. Example of image processing according to second embodiment

3-3. Image processing procedure according to second embodiment

4. Third embodiment

4-1. Configuration of image processing system according to thirdembodiment

5. Other embodiments

6. Hardware configuration

1. First Embodiment

[1-1. Configuration of Image Processing System According to FirstEmbodiment]

FIG. 1 is a diagram illustrating an example of an image processingsystem 1 according to a first embodiment of the present disclosure. Asillustrated in FIG. 1, the image processing system 1 includes an HMD 10and an image processing apparatus 100.

The HMD 10 is a display device mounted on the user's head and is alsoreferred to as a wearable computer. The HMD 10 realizes displayprocessing according to the orientation and movement of the user's body,moving speed, or the like.

The image processing apparatus 100 is an information processing devicethat executes image processing according to the present disclosure. Theimage processing apparatus 100 transmits the content held in theapparatus to the HMD 10 in response to a request transmitted from theHMD 10, for example.

The image processing apparatus 100 transmits, as content, for example,spherical content captured by a spherical camera capable of 360-degreeomnidirectional imaging. In general, spherical content has a largeramount of information compared with the volume of information (imagesize) a display device such as the HMD 10 can display at one time. Thatis, since the HMD10 cannot display the entire area of the sphericalcontent at the same time, the HMD10 displays only a partial area cut outaccording to the display size of the display device (in other words, theviewing angle of the user).

Here, due to a characteristic of spherical content created byomnidirectional imaging in 360 degrees, it is difficult to avoidsimultaneously capturing both bright and dark locations into one image.For this reason, it is desirable that the spherical content be recordedwith a wider dynamic range than ordinary content with a limited imagingrange (for example, content created under an environment usingadjustable exposure). For example, spherical content is preferablyrecorded in high dynamic range (HDR) or the like, which is a wider thanstandard dynamic range (SDR), which is an ordinary dynamic range.

However, the viewing environment for spherical content, such as HMD10,does not always support reproduction of the content recorded in HDR. Inthis case, the spherical content recorded in HDR is converted into alower dynamic range such as SDR at the time of reproduction inaccordance with the viewing environment such as the HMD10. At this time,performing conversion to SDR uniformly for entire areas of the sphericalcontent means applying the conversion process on areas having mixture ofbrightness/darkness expressions, leading to a failure in obtaining aclear image. Specifically, performing conversion to SDR uniformly onentire areas of spherical content would lead to occurrence of blown-outhighlights or crushed shadows, or lead to an artificial expression ofareas including a mixture of brightness/darkness expressions so as toresult in conversion into an unnatural image.

To handle these, the image processing system 1 according to the presentdisclosure extracts an area corresponding to the viewing angle of theuser of the HMD 10 from the spherical content and performs interactiveconversion process according to the distribution of the luminance valuesof the pixels included in the extracted area. With these processes, theimage processing system 1 according to the present disclosure providesthe user with a converted image in which brightness/darkness expressionis appropriately achieved according to the visual characteristics of theuser. Hereinafter, each of devices included in the image processingsystem 1 according to the present disclosure will be described withreference to FIG. 1.

First, the HMD 10 will be described. As illustrated in FIG. 1, the HMD10 includes processing units such as a detector 15, a transmitting unit16, a receiving unit 17, and a display control unit 18. Each of theprocessing units is implemented by execution of programs stored in theHMD 10 by a central processing unit (CPU), a micro processing unit(MPU), or the like, using random access memory (RAM) or the like, as aworking area. In addition, each of the processing units may beimplemented by an integrated circuit such as an application specificintegrated circuit (ASIC) or a field programmable gate array (FPGA).

The detector 15 controls a sensor 15A included in the HMD 10 to detectvarious types of information regarding the motion of a user, such as thebody orientation, inclination, movement, and moving speed of the user.Specifically, the detector 15 detects information regarding the user'shead and posture, movement of the user's head and body (acceleration andangular velocity), the direction of the visual field, the speed ofviewpoint movement, or the like, as the information regarding the motionof the user. For example, the detector 15 controls various motionsensors as the sensor 15A, such as a triaxial acceleration sensor, agyro sensor, and a speed sensor, so as to detect information regardingthe motion of the user. Note that the sensor 15A need not be providedinside the HMD 10 and may be an external sensor connected to the HMD 10with wired or wireless connection, for example.

The transmitting unit 16 transmits various types of information via awired or wireless network or the like. For example, the transmittingunit 16 transmits information regarding user's motion detected by thedetector 15 to the image processing apparatus 100. Furthermore, thetransmitting unit 16 transmits to the image processing apparatus 100 arequest to transmit spherical content to the HMD 10.

The receiving unit 17 receives various types of information via a wiredor wireless network. For example, the receiving unit 17 receives animage displayed by the display control unit 18 (more specifically, datasuch as pixel information displayed as an image). Specifically, thereceiving unit 17 receives an image having a partial area converted toSDR by the image processing apparatus 100, out of the spherical content.

The display control unit 18 controls the display process of the imagereceived by the receiving unit 17. Specifically, the display controlunit 18 controls a process of displaying an image having a partial areaconverted into the SDR by the image processing apparatus 100, out of thespherical content. Specifically, the display control unit 18 controlsthe process of displaying the image converted into SDR onto a display18A. The display 18A is implemented by an organic electro-luminescence(EL) display or a liquid crystal display, for example.

Although not illustrated in FIG. 1, the HMD 10 may include an input unitfor receiving an operation from a user, a storage unit that storesreceived spherical content, an output unit having a voice outputfunction, or the like.

Next, the image processing apparatus 100 will be described. Asillustrated in FIG. 1, the image processing apparatus 100 includesprocessing units such as a storage unit referred to as a content storageunit 121, an acquisition unit 131, an area determination unit 132, ananalyzing unit 133, a variable determination unit 134, a converter 135,and a transmitting unit 136.

The content storage unit 121 is implemented by semiconductor memoryelements such as random access memory (RAM) and flash memory, or storagedevices such as a hard disk or an optical disk. The content storage unit121 stores spherical content recorded in HDR, for example. The sphericalcontent includes video content such as still images and movies createdby imaging with a spherical camera or stitching a plurality of images.

Each of the processing units such as the acquisition unit 131 isimplemented by execution of programs stored in the image processingapparatus 100 (for example, an image processing program recorded in arecording medium according to the present disclosure) by the CPU, MPU,or the like, using RAM or the like as a working area. Furthermore, eachprocessing unit may be implemented by an integrated circuit such as ASICor FPGA.

The acquisition unit 131 acquires various types of information via awired or wireless network or the like. For example, the acquisition unit131 acquires the information regarding the user's motion transmittedfrom the HMD 10. Furthermore, the acquisition unit 131 receives arequest transmitted from the HMD 10 and containing a request to transmitthe spherical content to the HMD 10.

Based on the information regarding the user's motion acquired by theacquisition unit 131, the area determination unit 132 determines apredetermined area displayed on the display control unit 18 of the HMD10(hereinafter, referred to as a “processing area” for distinction) out ofthe spherical content to be transmitted to the HMD10.

For example, the area determination unit 132 specifies an area in thespherical content that the user intends to view, based on theinformation regarding the head posture of the user wearing the HMD 10.Specifically, the area determination unit 132 specifies an area in thespherical content corresponding to the user's visual field (hereinafter,referred to as a “display area” for distinction) based on theinformation detected by the sensor 15A included in the HMD10. Note thatvarious known techniques may be appropriately used for such a displayarea specifying process. Furthermore, the information regarding user'smotion may be not only the information transmitted from the HMD10 (forexample, the visual field direction estimated from the user's head andposture, or the user's motion speed) but also information calculated orestimated based on the information transmitted from the HMD10. Forexample, the area determination unit 132 may calculate coordinateinformation for specifying the display area from the spherical contentbased on the information transmitted from the HMD 10. In this case, theinformation regarding user's motion may also be rephrased as thecoordinate information for specifying a display area.

Subsequently, the area determination unit 132 determines a processingarea corresponding to the actual viewing environment from the displayarea specified on the spherical content. For example, the areadetermination unit 132 converts the display area specified on the imageforming the spherical content (for example, equirectangular projectionimage, cube map image, or fisheye lens image) into a perspectiveprojection image appropriate for the viewing angle of the HMD 10.Subsequently, the area determination unit 132 determines the perspectiveprojection image obtained by conversion as a processing area on the HMD10. In other words, the area determination unit 132 performs projectionconversion of the display area being a part of the spherical contentspecified based on the information regarding user's motion so as togenerate a post-projection conversion image. Subsequently, the areadetermination unit 132 determines the area corresponding to thepost-projection conversion image generated by the projection conversion,as a processing area.

The analyzing unit 133 analyzes information regarding individual pixelsincluded in the processing area determined by the area determinationunit 132 (that is, the image obtained after projection conversion of thedisplay area). Specifically, the analyzing unit 133 calculates the totalnumber of pixels included in the processing area. The analyzing unit 133also acquires the luminance values of individual pixels included in theprocessing area.

Here, the process performed at the time of conversion of an image fromHDR to SDR is referred to as tone mapping that weights and roundsluminance values of the pixels of the HDR based on a predeterminedluminance value. Hereinafter, the tone mapping process will be describedstep by step.

First, the analyzing unit 133 analyzes pixels within the processing areato calculate a luminance value to be used as a reference for tonemapping (hereinafter, referred to as “reference luminance”). Amongvarious conventional techniques present for tone mapping, a method usinga non-linear conversion function that allocates a large volume ofinformation to low luminance and enhances compression of high luminanceis generally considered to be closer to human visual characteristics. Aknown example of a conversion formula of tone mapping corresponding tosuch visual characteristics is Formula (1).

$\begin{matrix}{{\overset{\_}{L}}_{\omega} = {\frac{1}{N}{\exp \left( {\Sigma_{x,y}\mspace{14mu} {\log \left( {\delta + {L_{\omega}\left( {x,y} \right)}} \right)}} \right)}}} & (1)\end{matrix}$

In Formula (1), L_(ω)(x,y) represents a luminance value at an arbitrarypixel ω(x,y) included in a processing area. N represents the totalnumber of pixels within the processing area. Furthermore, δ is anarbitrary constant for avoiding a singular point when a black pixel ispresent in the image, for example. As illustrated in Formula (1), L⁻_(ω) is the logarithmic mean of the luminance values of all pixels inthe processing area, and this luminance value L⁻ _(ω) is defined as thereference luminance for tone mapping in the processing area.

Tone mapping is performed by scaling the luminance values of individualpixels recorded in HDR using the reference luminance calculated byFormula (1). For example, scaling is performed using Formula (2).

$\begin{matrix}{{L\left( {x,y} \right)} = {\frac{a}{{\overset{\_}{L}}_{\omega}}{L_{\omega}\left( {x,y} \right)}}} & (2)\end{matrix}$

In Formula (2), L(x,y) represents the scaled luminance of the arbitrarypixel ω(x,y). That is, the luminance value L_(ω) of the pixel recordedin HDR is scaled to the luminance value L of the pixel displayed in SDRthrough the process of Formula (2).

The value “a” in the Formula (2) represents a variable generallyreferred to as a key value, being a variable that determines thebrightness of the entire image. Examples of conventionally set keyvalues include “0.045”, “0.09”, “0.18”, “0.36”, or “0.72”. The largerthe key value, the higher the luminance value of the image displayed inthe processing area, enabling display of an image with higherbrightness. Note that in the image processing of the present disclosure,as will be described below in detail, the key value a in Formula (2) isnot set to a conventional value but is determined by the variabledetermination unit 134 based on the distribution of the luminance valuesof individual pixels included in the processing area.

The luminance value scaled using Formula (2) is further constrained toenhance the compression of the high luminance values. For example,Formula (3) is further applied to the scaled luminance value L(x, y).

$\begin{matrix}{{L_{d}\left( {x,y} \right)} = \frac{L\left( {x,y} \right)}{1 + {L\left( {x,y} \right)}}} & (3)\end{matrix}$

In Formula (3), L_(d)(x,y) represents the luminance value of the pixel(x,y) after SDR conversion. In the present disclosure, the luminancevalue is represented by an arbitrary numerical value in a range 0 to 1in both HDR and SDR using a representation method such as a floatingpoint.

In this manner, the analyzing unit 133 specifies the display area basedon the information regarding the motion of a user and further determinesthe reference luminance L⁻ _(ω) for the tone mapping from individualpixels included in the processing area obtained by perspectiveprojection of the display area.

The variable determination unit 134 determines a variable used tocalculate the luminance values of individual pixels at the time ofconversion of the first dynamic range, based on the distribution of theluminance values of individual pixels included in the processing area,out of the first video data recorded in the first dynamic range. In thepresent disclosure, the first dynamic range represents HDR. In addition,the first video data represents spherical content.

That is, the variable determination unit 134 determines a variable usedfor calculating the luminance values of individual pixels when HDR isconverted into SDR, based on the distribution of the luminance values ofindividual pixels included in the processing area. In the presentdisclosure, the variable determined by the variable determination unit134 is the key value a illustrated in Formula (2).

For example, the variable determination unit 134 specifies the mode ofthe luminance values in the processing area based on the distribution ofindividual pixels included in the processing area. Subsequently, thevariable determination unit 134 determines the key value a so that thepixels in the vicinity of the luminance of the mode are expressed inhigher precision in the converted image. In other words, the variabledetermination unit 134 determines the key value a so that thebrightness/darkness expression near the mode is represented in thecontrast with higher precision in the converted image.

As an example, the variable determination unit 134 dynamicallydetermines the key value a so that a converted luminance value L_(d) ofthe pixel corresponding to the mode is set to “0.5”. This is becausesetting L_(d)(x, y) to “0.5” increases the bit allocation (informationvolume) for expressing the luminance values near the mode.

This will be described with reference to FIGS. 2 and 3. FIG. 2 is ahistogram illustrating a distribution of luminance values of pixelsincluded in an image. FIG. 2 conceptually illustrates a distribution ofluminance values of the pixels included in the image (processing area).In the histogram illustrated in FIG. 2, the vertical axis corresponds tothe number of pixels, while the horizontal axis corresponds to theluminance value. As illustrated in FIG. 2, a relatively large number ofpixels among all the pixels included in the processing area are oftendistributed in the vicinity of a mode L_(m) of the luminance values, intypical cases.

Subsequently, in a case where the variable for individual pixelsillustrated in FIG. 2 is dynamically determined so that the convertedvalue of the pixel corresponding to the mode L_(m) (that is, L_(d)(x,y)of Formula (3)) is set to “0.5”, Formula (3) will be drawn as a graphillustrated in FIG. 3. FIG. 3 is a diagram illustrating an example ofimage processing according to the first embodiment of the presentdisclosure. In the graph of FIG. 3, the vertical axis corresponds to thefinal pixel luminance value L_(d) calculated by Formula (3), while thehorizontal axis corresponds to the pixel luminance value L immediatelyafter being scaled by Formula (2). As illustrated in FIG. 3, whenassuming that the left side of Formula (3) in the mode L_(m) is “0.5”, arelatively large amount of information would be allocated in thevicinity of the mode L_(m).

For example, in the example illustrated in FIG. 3, a range E01 and arange E02 of the luminance value before conversion by Formula (3)indicate a same luminance value range (width). In contrast, a convertedluminance value range F01 corresponding to the range E01 includes awider range of luminance values than a converted luminance value rangeF02 corresponding to the range E02. This means that pixels havingluminance values in the vicinity of the mode L_(m) are assigned to finergradations in SDR conversion, leading to high-claritybrightness/darkness expression in the converted image.

As described above, assuming that L_(d)(x,y) in Formula (3) is “0.5”,the key value a is expressed as follows from the Formulas (2) and (3).

$\begin{matrix}{a = \frac{{\overset{\_}{L}}_{\omega}}{L_{m}}} & (4)\end{matrix}$

In Formula (4), “L_(m)” indicates the mode of the luminance values ofeach of images included in the processing area. In this manner, thevariable determination unit 134 can dynamically determine the key valuea, which is a variable that determines the brightness of the convertedimage, based on the distribution of the luminance values of individualpixels included in the processing area. Specifically, the variabledetermination unit 134 determines a value calculated by dividing thelogarithmic mean of the luminance value (reference luminance L⁻ _(ω)) ofindividual pixels by the mode L_(m) of the luminance values ofindividual pixels included in the processing area, as the key value a.

While Formula (4) illustrates a method of calculating the key value a inthe case where the converted luminance value L_(d)(x,y) of the modeL_(m) is assumed to be “0.5”, L_(d)(x,y) does not necessarily have to be“0.5”. For example, in order to appropriately adjust the distribution ofindividual pixels included in the conversion target processing area andthe brightness of the entire image after conversion, the variabledetermination unit 134 may set the luminance value L_(d)(x,y) afterconversion of the mode L_(m) to a value other than “0.5”.

Furthermore, the variable determination unit 134 may determine the keyvalue a in a case where the information regarding the user's motionsatisfies a predetermined condition. The variable determination unit 134may skip the process of newly determining the key value a in a casewhere the information regarding the user's motion does not satisfy thepredetermined condition.

The information regarding user's motion includes information fordetermining the processing area, represented by the informationregarding the head posture of the user and the motion speed of the user,for example. Furthermore, the predetermined condition is, for example, acondition that a user's motion speed or the like should satisfy.

For example, the variable determination unit 134 may make adetermination of newly determining the key value a in a case where theuser's motion speed is a predetermined threshold or less, as a casewhere a predetermined condition is satisfied. Specifically, in a casewhere the user's motion speed is a predetermined threshold or less, thevariable determination unit 134 determines the key value a based on thedistribution of the luminance values of individual pixels included inthe processing area determined according to the information regardingthe user's motion. The user's motion speed may be various values such asthe motion speed of the HMD 10 or the angular speed of the user's head.For example, the user's motion speed may be an arbitrary speed at whichthe user is estimated to be gazing at a certain area of the sphericalcontent. That is, when the user's motion speed exceeds a predeterminedthreshold, it is assumed that the user cannot decide which area of thespherical content to view. In contrast, when the user's motion speed isa predetermined threshold or less, it is assumed that the user is gazingat the area of the spherical content to which the line of sight isdirected. Subsequently, the variable determination unit 134 executes theprocess of determining the key value a in a state where it can bedetermined that the area the user is gazing at is decided to someextent.

In contrast, it is assumed that the area the user is gazing at is notdecided when the user's motion speed exceeds the predeterminedthreshold, and thus, the variable determination unit 134 does not needto newly determine the key value a. In a case where the process of newlydetermining the key value a is skipped, the converter 135 describedbelow performs the conversion process using an existing key value adetermined based on the distribution of the luminance values ofindividual pixels included in the processing area that satisfies apredetermined condition. For example, the converter 135 performs theconversion process using the key value a determined immediately beforeor a predetermined value (initial value or the like) that has beenpreliminarily set. In other words, the converter 135 performs theconversion process by using the existing key value a determined beforethe determination time point of the above-described predeterminedcondition.

This is because a case where the speed exceeding a predeterminedthreshold is observed indicates an assumable situation in which the useris moving around actively or looking around various items of sphericalcontent. Allowing the image processing apparatus 100 to newly calculatethe key value a and continue the conversion process in real timeaccording to the line-of-sight of the user regardless of such asituation might cause occurrence of flicker in the image displayed onthe HMD 10. For this reason, in the case where the user's motion speedexceeds a predetermined threshold, the variable determination unit 134would not newly determine the key value a to suppress switching of thetone mapping process, thereby achieving prevention of flicker. Note thatthe threshold of the motion speed is not necessarily a constant valueand may be appropriately varied depending on the scene being reproducedin the spherical content or the details of the content.

Furthermore, the variable determination unit 134 may determine whetheror not to newly determine the key value a on the condition other thanthe motion speed of the user. For example, the variable determinationunit 134 may newly determine the key value a in a case where pixelinformation of individual pixels included in the processing areasatisfies a predetermined condition. Specifically, in a case where anamount of change in the pixel information of individual pixels includedin the processing area in a predetermined time is a predeterminedthreshold or less, as a case where a predetermined condition issatisfied, the variable determination unit 134 may determine the keyvalue a based on the distribution of the luminance values of individualpixels included in the processing area. The variable determination unit134 may skip the process of newly determining the key value a in a casewhere the pixel information of individual pixels included in theprocessing area does not satisfy the predetermined condition. In a casewhere the determination process of the key value a is skipped, theconverter 135 described below performs the conversion process using anexisting key value a determined before the determination time point ofthe predetermined condition regarding the pixel information ofindividual pixels included in the processing area.

For example, in a case where the spherical content is a movie, it isassumed that the pixel information (luminance or color information) ofindividual pixels in the processing area frequently changes within apredetermined time (for example, several seconds). Frequently updatingthe key value a and the reference luminance in such a case might causeoccurrence of flicker similar to the above case. To handle this, thevariable determination unit 134 may determine whether or not the amountof change in the pixel information of individual pixels included in theprocessing area during a predetermined time is a predetermined thresholdor less and may newly determine the key value a in a case where theamount of change is the predetermined threshold or less, for example.

The converter 135 converts the first video data into second video datadisplayed in the second dynamic range, based on the variable determinedby the variable determination unit 134. In the present disclosure, thesecond dynamic range represents SDR, for example. Furthermore, thesecond video data represents an image corresponding to the processingarea, being an image displayed in SDR on the HMD 10 or the like.

Specifically, the converter 135 converts the spherical content recordedin HDR into an image displayed in SDR based on the reference luminance Ldetermined by the analyzing unit 133 and based on the key value adetermined by the variable determination unit 134.

As described above, the reference luminance L⁻ _(ω) and the key value aare dynamically determined for each processing areas determinedaccording to the user's motion. Therefore, for example, in a case wherean area in which a relatively bright subject is captured (area whereoverall luminance value is high) matches the viewing angle, thereference luminance L becomes high, and therefore bit allocations areshifted to the high luminance side in execution of tone mapping. In thiscase, the converter 135 can convert an image recorded in HDR into an SDRimage in which blown-out highlights are suppressed.

In contrast, in a case where an area in which a dark subject is captured(area where overall luminance value is low) matches the viewing angle,the reference luminance L⁻ _(ω) becomes low, and therefore bitallocations are shifted to the low luminance side in execution of tonemapping. In this case, the converter 135 can convert an image recordedin HDR into an SDR image in which crushed shadows are suppressed.

The transmitting unit 136 transmits the spherical content converted intoSDR by the converter 135 to the HMD 10. For example, the transmittingunit 136 transmits, to the HMD 10, image data of a portion correspondingto the processing area determined according to the user's motion, out ofthe spherical content.

[1-2. Example of Image Processing According to First Embodiment]

Image processing executed by the image processing system 1 illustratedin FIG. 1 will be specifically described with reference to FIG. 4. FIG.4 is a diagram (1) illustrating an example of image processing accordingto the first embodiment of the present disclosure. An equirectangularprojection image 50 illustrated in FIG. 4 is an example of an imageincluded in the spherical content recorded in HDR.

First, the HMD 10 detects the motion of the user wearing the HMD 10.Subsequently, the HMD 10 transmits information regarding the detectedmotion of the user to the image processing apparatus 100. The imageprocessing apparatus 100 specifies a display area 51 based on theinformation regarding the user's motion acquired from the HMD 10 (stepS11).

The image processing apparatus 100 performs perspective projectionconversion on the image corresponding to the display area 51 todetermine the processing area corresponding to the display area 51 (stepS12). In the example of FIG. 4, the image processing apparatus 100determines a processing area 52 as the display area corresponding to thedisplay area 51. The processing area 52 indicates an area in the HMD 10that corresponds to the viewing angle of a user when the user actuallyviews the image.

Furthermore, the image processing apparatus 100 converts the image ofthe processing area 52 into SDR based on the distribution of individualpixels included in the processing area 52 (step S13). That is, the imageprocessing apparatus 100 performs the conversion process by using theinformation of individual pixels included in the processing area 52corresponding to the image after the projection conversion, rather thanusing the display area 51 cut out from the equirectangular projectionimage 50. In this manner, the image processing apparatus 100 performsconversion process using pixel information included in the area(processing area) that the user actually views on the HMD 10, enablingproviding a converted image as more natural expression to the user.Specifically, the image processing apparatus 100 dynamically determinesthe key value a by using the pixel information included in the areaactually viewed by the user and performs conversion using the determinedkey value a, making it possible to provide an image achieving abrightness/darkness expression to match the visual characteristics ofhuman.

Here, in the example of FIG. 4, the logarithmic mean or the mode of theluminance values of the pixels included in the processing area 52 isassumed to be a higher value compared with the logarithmic mean or themode of the luminance values of the pixels included in the entireequirectangular projection image 50. In other words, the processing area52 is an image that is brighter as compared to the entireequirectangular projection image 50. The image processing apparatus 100performs tone mapping using the key value a determined based on thelogarithmic mean luminance value or the mode of individual pixelsincluded in the processing area 52 so as to perform conversion into SDR.As a result, as illustrated in FIG. 4, the image processing apparatus100 can generate a converted image 53 that has been converted to improvevisibility of brightness/darkness expression of the pixels havingblown-out highlights in the processing area 52.

Subsequently, when the user moves the viewing field to view the areaother than the display area 51, the HMD 10 detects the movement of thevisual field of the user. Subsequently, the HMD 10 transmits to theimage processing apparatus 100 information regarding the motion of theuser who wears the HMD 10. The image processing apparatus 100 specifiesa display area 61 based on the information regarding the user's motionacquired from the HMD 10 (step S21).

The image processing apparatus 100 performs perspective projectionconversion on the image corresponding to the display area 61 todetermine the processing area corresponding to the display area 61,similarly to the process in the display area 51 (step S22). In theexample of FIG. 4, the image processing apparatus 100 determines aprocessing area 62 as the processing area corresponding to the displayarea 61.

Furthermore, the image processing apparatus 100 converts the imagecorresponding to the processing area 62 into SDR based on thedistribution of individual pixels included in the processing area 62(step S23). In the example of FIG. 4, the logarithmic mean or the modeof the luminance values of the pixels included in the processing area 62is assumed to be a lower value compared with the logarithmic mean or themode of the luminance values of the pixels included in the entireequirectangular projection image 50. In other words, the processing area62 is an image that is darker as compared to the entire equirectangularprojection image 50. The image processing apparatus 100 performs tonemapping using the key value a determined based on the logarithmic meanluminance value or the mode of individual pixels included in theprocessing area 62 so as to perform conversion into SDR. As a result, asillustrated in FIG. 4, the image processing apparatus 100 can generate aconverted image 63 that has been converted to improve visibility ofbrightness/darkness expression of the pixels having crushed shadows inthe processing area 62.

The above process will be described in detail with reference to FIG. 5.FIG. 5 is a diagram (2) illustrating an example of image processingaccording to the first embodiment of the present disclosure. In step S13according to FIG. 4, the graph corresponding to Formula (3) isillustrated by a curve 54 illustrated in FIG. 5, for example.Furthermore, in step S23 according to FIG. 4, the graph corresponding toFormula (3) is illustrated by a curve 64 illustrated in FIG. 5, forexample. In the example of FIG. 4, in comparison between the processingarea 52 and the processing area 62, the processing area 62 is assumed tohave a lower luminance distribution as a whole. Therefore, a mode L_(m1)of the luminance values in the processing area 62 is lower than a modeL_(m2) of the luminance values in the processing area 52. As describedin FIG. 3, since the mode L_(m1) and the mode L_(m2) has the value of“0.5” as the converted luminance value, the curve 64 has a higher rateof increase (slope) in a relatively low range of the luminance value L,as compared to the curve 54, as illustrated in FIG. 5.

This means that in the conversion process of step S23, a larger volumeof information (bits) is allocated to the relatively low luminance valuerange, as compared with the conversion process of step S13. That is, theimage processing apparatus 100 performs the conversion process with moreweight on the portion with a dark display level (low luminance value) inthe processing area 62. This enables the image processing apparatus 100to generate the converted image 63 in which the portion of theprocessing area 62 having a dark display level is displayed withhigh-clarity brightness/darkness expression. On the other hand, in theconversion process of step S13, a larger volume of information isallocated to the range of relatively high luminance value, as comparedwith the conversion process of step S23. That is, the image processingapparatus 100 performs the conversion process with more weight on theportion with a bright display level (high luminance value) in theprocessing area 52. This enables the image processing apparatus 100 togenerate the converted image 53 in which the portion of the processingarea 52 having a bright display level (for example, a portion with theluminance value having a possibility of blown-out highlights) isdisplayed with high-clarity brightness/darkness expression.

[1-3. Procedure of Image Processing According to First Embodiment]

Next, an image processing procedure according to the first embodimentwill be described with reference to FIG. 6. FIG. 6 is a flowchartillustrating a flow of a process according to the first embodiment ofthe present disclosure.

As illustrated in FIG. 6, the image processing apparatus 100 acquiresinformation regarding a user's motion from the HMD 10 (step S101).Subsequently, the image processing apparatus 100 specifies an areacorresponding to the user's motion (for example, the user's visual fielddirection) within one image of the spherical content, based on theinformation regarding the user's motion (step S102). The area specifiedin step S102 is, for example, the display area 51 or the display area 61illustrated in FIG. 4.

Subsequently, the image processing apparatus 100 reads out the sphericalcontent recorded in HDR from the content storage unit 121 (step S103).Furthermore, the image processing apparatus 100 performs projectionconversion on the area specified in step S102 to determine a processingarea corresponding to the area (step S104). The processing areadetermined in step S104 corresponds to the processing area 52 or theprocessing area 62 illustrated in FIG. 4, for example.

Thereafter, the image processing apparatus 100 determines (step S105)whether or not the user's motion speed is a predetermined threshold orless based on the information regarding the user's motion acquired instep S101. In a case where the user's motion speed is the predeterminedthreshold or less (step S105; Yes), the image processing apparatus 100analyzes the pixel information of individual pixels in the processingarea (step S106). Subsequently, the image processing apparatus 100determines a variable (key value a) used for tone mapping based on theanalysis result of individual pixels (step S107).

In contrast, in a case where the user's motion speed exceeds thepredetermined threshold (step S105; No), the image processing apparatus100 reads out the already determined reference luminance and variable(step S108). In other words, the image processing apparatus 100 skipsthe process of newly determining (calculating) the reference luminanceor variables. In this case, the image processing apparatus 100 uses thereference luminance and the variable determined in the immediatelypreceding process or uses the reference luminance and the variable of apreset prescribed value. That is, the image processing apparatus 100uses the existing variable that has been determined before the timepoint (in the example of FIG. 6, step S105) of determining whether ornot to perform variable determination.

The image processing apparatus 100 converts (step S109) an image in theprocessing area from HDR to SDR based on the reference luminance and thevariable determined in step S107 or step S108. Subsequently, the imageprocessing apparatus 100 transmits the converted image to the HMD 10(step S110).

In this manner, the image processing apparatus 100 according to thefirst embodiment performs image processing that selectively uses pixelsincluded in the processing area determined according to the user'smotion, rather than using the pixel information of the entire sphericalcontent. Specifically, the image processing apparatus 100 according tothe first embodiment dynamically determines the key value a and thereference luminance used in converting the video data, based on thedistribution of the luminance values of individual pixels included inthe processing area. With this process, the image processing apparatus100 according to the first embodiment can perform the conversion processsuitable to the image that the user intends to view, leading togeneration of the converted image achieving a high-claritybrightness/darkness expression of the image. Furthermore, the imageprocessing apparatus 100 according to the first embodiment performs aprocess of newly determining a variable in a case where the user'smotion speed is a threshold or less, making it possible to preventoccurrence of flicker or the like.

2. Modification of First Embodiment

The first embodiment is an example in which the image processingapparatus 100 performs the process of determining the referenceluminance and the key value a in the processing area based on theinformation regarding the user's motion in a case where the user'smotion speed falls to a predetermined threshold or less. Here, when themotion speed falls to a threshold or below, the image processingapparatus 100 may perform processing of gradually changing the referenceluminance and the key value a determined immediately before to a newreference luminance and key value a, rather than immediately determiningthe new reference luminance and the key value a. That is, the imageprocessing apparatus 100 performs the conversion process over time so asto gradually transition to the converted image, rather than immediatelyconvert the processing area. Accordingly, the image processing apparatus100 can perform the conversion process that simulates human visualadaptation (dark adaptation or light adaptation) which occurs when ahuman moves from a bright place to a dark place, or the like, making itpossible to provide the user with more realistic and natural image.

That is, the variable determination unit 134 according to themodification of the first embodiment determines a first variable (keyvalue a based on the processing area before the change) and thereafterdetermines a second variable (based on the processing area after thechange. During these processes, the variable determination unit 134determines values to achieve stepwise transition from the first variableto the second variable. Furthermore, the converter 135 according to themodification of the first embodiment converts the first video data bysequentially using the values to achieve stepwise transition from thefirst variable to the second variable, determined by the variabledetermination unit 134.

The above process will be described with reference to FIG. 7. FIG. 7 isa diagram illustrating an example of image processing according to amodification of the first embodiment of the present disclosure. FIG. 7illustrates an example in which the image processing apparatus 100gradually changes the reference luminance and the key value a determinedimmediately before to the new reference luminance and the key value a soas to perform tone mapping in accordance with passage of time.

Similar to the example illustrated in FIG. 4, the image processingapparatus 100 in FIG. 7 specifies the display area 51 based on theinformation regarding the user's motion detected by the HMD 10 (stepS31).

The image processing apparatus 100 performs perspective projectionconversion on the image corresponding to the display area 51 todetermine the processing area 52 corresponding to the display area 51(step S32).

Subsequently, the image processing apparatus 100 determines thereference luminance and the key value a in the processing area 52,similarly to FIG. 4. At this time, the image processing apparatus 100dynamically determines the reference luminance and the key value adefined in accordance with passage of time by using Formulas (5) and(6), for example.

$\begin{matrix}{{\overset{\_}{L}}_{\omega} = {{\overset{\_}{L}}_{pre} + {\Delta \; {L(t)}}}} & (5) \\{{\Delta \; {L(t)}} = {\frac{{\overset{\_}{L}}_{cur} - {\overset{\_}{L}}_{pre}}{T}t}} & (6)\end{matrix}$

In Formula (5), L⁻ _(pre) represents the logarithmic mean luminancevalue determined in the immediately preceding determination process,that is, the previous reference luminance. Furthermore, t in Formula (5)indicates the time. Furthermore, in Formula (5), ΔL(t) is a functionindicating the luminance value that changes over time. Furthermore, inFormula (6), L⁻ _(cur) indicates the logarithmic mean luminance valuecorresponding to the processing area 52 determined based on theinformation regarding the current user's motion, that is, the referenceluminance for the current time point. Furthermore, T in the Formula (6)is an arbitrary constant for determining the amount of change in thetime direction.

As illustrated in Formula (5), the reference luminance L⁻ _(ω) accordingto the modification is determined based on the reference luminance L⁻_(pre) determined immediately before and the amount of change ΔL(t) overtime. Subsequently, the image processing apparatus 100 converts theimage of the processing area 52 using the reference luminance L⁻ _(ω) inFormula (5) (step S33). With this process, in a case where the referenceluminance is determined, the image in the processing area 52 isconverted through an intermediate image 57 and an intermediate image 58as illustrated in FIG. 7 to the image 53, rather than immediatelyconverted into the converted image 53. In this case, the key value aalso changes in accordance with the change in the reference luminance,as illustrated in the above Formula (4).

In this manner, in the modification of the first embodiment, the imageprocessing apparatus 100 performs tone mapping using the referenceluminance and the key value a that change over time. With thisconfiguration, the image processing apparatus 100 can perform theconversion process from HDR to SDR by a natural expression thatsimulates the light adaptation and dark adaptation characteristics inhuman vision.

Incidentally, it is known that the time needed for dark adaptation islonger than the time needed for light adaptation in human vision.Therefore, the image processing apparatus 100 may change the value of Tin Formula (6) in accordance with the relationship between L⁻ _(pre) andL⁻ _(cur). Specifically, the image processing apparatus 100 may set thevalue of T relatively large in a case where the relationship of L⁻_(pre)>L⁻ _(cur) is established, that is, when simulating darkadaptation. In this case, the image processing apparatus 100 may alsoset a relatively long time for the time taken before the value L⁻ _(ω)illustrated in Formula (5) finally converges to the value L⁻ _(cur).Furthermore, in a case where the relationship of L⁻ _(pre)<L⁻ _(cur) isestablished, that is, when simulating light adaptation, the imageprocessing apparatus 100 may set the value T to a value smaller than thevalue adopted in simulating dark adaptation. In this case, the imageprocessing apparatus 100 may also set the time taken before the value L⁻_(ω) illustrated in Formula (5) finally converges to the value L⁻ _(cur)to the value smaller than the value adopted when simulating darkadaptation (that is, set the value so as to converge to the value L⁻_(cur) in a shorter time).

In addition, ΔL(t) in Formula (6) is simply expressed by a changefunction in the linear time direction. However, depending on the humanvisual characteristics and various situations reproduced in the content,ΔL(t) may be expressed by the change function in the non-linear timedirection.

As described above, in a case where the key value a is newly determined,the image processing apparatus 100 performs a conversion process basedon a value included between the newly determined key value a and theexisting key value a that had been determined before determination ofthe key value a, which is determined in accordance with the passage oftime.

The conversion process performed in accordance with the passage of timeas described above can be triggered by various situations. For example,the image processing apparatus 100 may perform the above variabledetermination process and conversion process in a case where the user'smotion speed falls to a predetermined threshold or below. Furthermore,the image processing apparatus 100 may perform the variabledetermination process and the conversion process in accordance with thetransition of the displayed scene, regardless of the user's motion. Forexample, when the spherical content is a movie, the pixel information ofthe image displayed on the HMD10 might change in some cases due to scenetransition (image change) even when there is no change in the motion ofthe user wearing the HMD10. In this case, the image processing apparatus100 newly calculates the key value a and the reference luminance inaccordance with the transition of the scene and performs the conversionprocess representing the light adaptation or the dark adaptation asdescribed above, making it possible to provide the user with a morenatural image. The image processing apparatus 100 may determine whetheror not the scene has transitions (in other words, whether or not toperform the conversion process according to the modification) based onthe occurrence of a change in the pixel information included in theimmediately preceding image beyond a predetermined threshold value, forexample. That is, the image processing apparatus 100 may perform theabove conversion process based on the greatness of the amount of changein the pixel information of individual pixels included in the processingarea beyond a predetermined threshold within a predetermined time.

In this manner, the variable determination unit 134 according to theimage processing apparatus 100 newly determines the key value a in acase where the information regarding the user's motion satisfies apredetermined condition or where the amount of change in the pixelinformation of individual pixels included in the processing areasatisfies a predetermined condition. Subsequently, in the case of newlydetermining the key value a, the variable determination unit 134determines a value that achieves a stepwise change with the passage oftime and that converges to the newly determined key value a. This valueis calculated, for example, by assigning the reference luminancecalculated by Formula (5) into Formula (4). Specifically, the variabledetermination unit 134 is a value that is included between the key valuea and the existing key value a that had been determined beforedetermination of the new key value a, which is determined in accordancewith passage of time. Subsequently, the converter 135 of the imageprocessing apparatus 100 converts the image corresponding to theprocessing area based on the value that converges to the newlydetermined key value a.

This enables the image processing apparatus 100 according to themodification of the first embodiment to perform a natural conversionprocess close to the characteristics of light adaptation and darkadaptation of human vision when converting the image recorded in HDR toSDR and displaying the image in SDR.

3. Second Embodiment

[3-1. Configuration of Image Processing System According to SecondEmbodiment]

Next, a second embodiment will be described. In the second embodiment,image processing that is executed when a virtual object other thanspherical content is displayed on the display 18A of the HMD 10 will bedescribed.

First, an image processing system 2 according to the second embodimentwill be described. FIG. 8 is a diagram illustrating an example of theimage processing system 2 according to the second embodiment of thepresent disclosure. As illustrated in FIG. 8, the image processingsystem 2 further includes a controller 20 as compared with the firstembodiment. The image processing apparatus 200 according to the secondembodiment further includes an analyzing unit 233, a variabledetermination unit 234, and a drawing unit 235. Note that description ofthe same configuration as the image processing apparatus 100 accordingto the first embodiment will be omitted.

The controller 20 is an information device connected to the imageprocessing apparatus 200 and the HMD 10 via a wired or wireless network.The controller 20 is an information device held and operated by the userwearing the HMD 10 so as to detect the movement of the user's hand andthe information input to the controller 20 by the user. Specifically, adetector 21 related to the controller 20 controls a sensor 21A (forexample, various motion sensors such as a triaxial acceleration sensor,a gyro sensor, or a speed sensor) to detect the three-dimensionalposition and speed of the controller 20. Subsequently, the detector 21transmits the detected three-dimensional position, speed, or the like,to the image processing apparatus 200. The detector 21 may transmit thethree-dimensional position of the device itself detected by an externalsensor such as an external camera.

The drawing unit 235 of the image processing apparatus 200 draws avirtual object so as to be superimposed onto the image displayed on theHMD 10 based on the information detected by the detector 21 of thecontroller 20. The drawing unit 235 draws a virtual object thatsimulates the user's hand in accordance with the user's actual motion(for example, hand movement), for example. Furthermore, in a case wherethe content displayed on the HMD 10 is a game, the drawing unit 235 maydraw a virtual object simulating a tool or the like used in the game.Note that various publicly known techniques may be appropriately usedfor such a virtual object drawing process.

In a case where a virtual object is included in the processing area, theanalyzing unit 233 and the variable determination unit 234 according tothe second embodiment perform analysis and variable determination usingthe luminance value set for the virtual object. Specifically, in a casewhere the pixels in the original image are hidden (superposed) by thevirtual object in the processing area, the analyzing unit 233 or thevariable determination unit 234 performs the process using the pixelinformation (such as the luminance value) of the virtual object.

For example, the variable determination unit 234 calculates the keyvalue a based on the luminance values of individual pixels included inthe processing area and based on the distribution of the luminancevalues of the virtual objects included in the processing area. Thevariable determination unit 234 uses a predetermined luminance value setin advance for the luminance value of the pixels forming the virtualobject. Furthermore, the variable determination unit 234 may determinethe luminance values of the pixels forming the virtual object, based onthe luminance value of the entire image of the spherical content onwhich the virtual object is superimposed. This enables the variabledetermination unit 234 to adjust the virtual object to be drawn in abright level when the spherical content is generally bright or adjustthe virtual object to be drawn in a dark level when the sphericalcontent is generally dark.

Furthermore, it is allowable to have a configuration in which thevariable determination unit 234 will calculate the key value a when thebehavior of the virtual object satisfies a predetermined condition andwould not newly calculate the key value a when the behavior of thevirtual object does not satisfy the predetermined condition.Specifically, in a case where the virtual object is drawn at a speedbeyond a predetermined speed by the user's operation on the controller20 at a speed beyond a predetermined threshold, the variabledetermination unit 234 may omit the calculation process of the key valuea in order to prevent occurrence of flicker on the screen, or the like.In contrast, in a case where the virtual object is drawn at apredetermined speed or less, the variable determination unit 234 maynewly determine the key value a so that the tone mapping can beperformed using the new key value a.

[3-2. Example of Image Processing According to Second Embodiment]

The image processing executed by the image processing system 2illustrated in FIG. 8 will be specifically described with reference toFIG. 9. FIG. 9 is a diagram illustrating an example of image processingaccording to the second embodiment of the present disclosure. With FIG.9, the image processing according to the second embodiment will bedescribed using an exemplary case where a virtual object is latersuperimposed on an image once converted from HDR to SDR. Theequirectangular projection image 50 illustrated in FIG. 9 is the sameimage as that illustrated in FIG. 4.

Similar to the example illustrated in FIG. 4, the image processingapparatus 200 specifies a display area 71 based on the informationregarding the user's motion detected by the HMD 10 (step S41).

The image processing apparatus 200 performs perspective projectionconversion of the image corresponding to the display area 71 todetermine a processing area 72 corresponding to the display area 71(step S42).

Furthermore, the image processing apparatus 200 converts the imagecorresponding to the processing area 72 into SDR based on thedistribution of individual pixels included in the processing area 72(step S43). The processing area 72 illustrated in FIG. 9 has blown-outhighlights in the upper center part but has relatively low luminancevalues in the other portions, resulting in a dark image as a whole. Incontrast, in a converted image 73 after the SDR conversion, the portionshaving a low luminance value and a low contrast level, such as the lowercentral part, have been changed to have a slightly increased brightnessand better clarity.

Here, it is assumed that the user performs an operation of superimposinga hand, which is a virtual object, on the converted image 73 via thecontroller 20. The image processing apparatus 200 derives a positionwhere a virtual object 80 is to be superimposed on the converted image73 based on the information detected by the controller 20.

Subsequently, the image processing apparatus 200 analyzes the imageincluding the converted image 73 and the pixels forming the virtualobject 80, as a processing target. Specifically, the image processingapparatus 200 calculates the reference luminance for tone mapping basedon the luminance values of individual pixels forming the virtual object80 and the luminance values of individual pixels included in theconverted image 73. Moreover, the image processing apparatus 200determines a key value a for tone mapping based on the luminance valuesof individual pixels forming the virtual object 80 and the distributionof the luminance values (mode of luminance value, etc.) of individualpixels included in the converted image 73.

Subsequently, the image processing apparatus 200 draws the virtualobject 80 so as to be superimposed onto the converted image 73, whileperforming tone mapping based on the luminance information of individualpixels for the case of including the virtual object 80 (step S44). Withthis process, the image processing apparatus 200 can display a convertedimage 74 on the HMD 10 and display the virtual object 80 superimposed onthe converted image 74.

In the converted image 74 illustrated in FIG. 9, since the upper centerpart of the converted image 73, which has blown-out highlights, ishidden by the virtual object 80, the distribution of the luminance valueof the entire image is smoothed, leading to the clearer contrast in thedisplay of the portion other than the virtual object 80. That is, theconverted image 74 is considered to be an image that simulates on thescreen a situation in which the visibility is improved at a locationwhere the visibility is deteriorated due to the influence of light, byblocking the light with the hand in the real world.

In the example illustrated in FIG. 9, the virtual object 80 is drawn inwhite for the purpose of explanation, but in reality, the luminancevalues of the pixels forming the virtual object 80 need not have highluminance values expressed in white. For example, when determining thekey value a, the variable determination unit 234 according to the secondembodiment may calculate a luminance value set to individual pixelsforming the virtual object 80 based on the luminance values ofindividual pixels included in the equirectangular projection image 50.For example, the luminance values of the pixels forming the virtualobject 80 may be the mean, the median, or the logarithmic mean luminancevalue, or the like of the luminance values of the pixels included in theequirectangular projection image 50. This enables the image processingapparatus 200 to appropriately set the luminance values of the pixelsforming the virtual object 80 to the value corresponding to thespherical content as a processing target, making it possible to preventthe execution of an unnatural conversion process due to superimpositionof the virtual object 80. Note that the luminance values of the pixelsforming the virtual object 80 are not necessarily the same luminancevalue but may be luminance values different from each other.

The process illustrated in FIG. 9 will be conceptually described withreference to FIG. 10. FIG. 10 is a diagram illustrating an example ofimage processing according to the second embodiment of the presentdisclosure. Similar to the histogram illustrated in FIG. 2, FIG. 10illustrates the number of pixels included in the processing area and thedistribution of luminance values.

For example, a curve 81 illustrated in FIG. 10 illustrates the number ofpixels and the distribution of the luminance values in the convertedimage 73 illustrated in FIG. 9. As illustrated in FIG. 9, the convertedimage 73 includes a relatively large number of pixels having a highluminance value (a portion expressed in white in FIG. 9) in the uppercenter part. Therefore, the curve 81 has a shape indicating that arelatively large number of pixels are distributed in a high luminancevalue range.

Here, as illustrated in FIG. 9, it is assumed that the user operates thecontroller 20 to superimpose the virtual object 80 onto the upper centerpart of the converted image 73 (step S51). In this case, the pixels thatare superimposed on the virtual object 80 out of the pixels included inthe converted image 73 are replaced with the luminance values of thepixels of the virtual object 80.

For example, a curve 82 illustrated in FIG. 10 denotes the distributionof the number of pixels and the luminance value when the virtual object80 is superimposed on the converted image 73 illustrated in FIG. 9. Asillustrated in the curve 82, after the virtual object 80 issuperimposed, the number of pixels having a high luminance valuedecreases as compared with the curve 81. Furthermore, as an effect ofsuperimposition of the virtual object 80, the mode of the luminancevalues changes from a mode L_(m3) to a mode L_(m4) in the luminancevalues on the curve 81.

In this case, the image processing apparatus 200 calculates thereference luminance and the key value a based on the distribution of theluminance values of individual pixels after the virtual object 80 issuperimposed and newly determines the reference luminance and the keyvalue a. Subsequently, the image processing apparatus 200 performs tonemapping on the converted image 73 based on the newly determinedreference luminance and the key value a, so as to generate the convertedimage 74.

[3-3. Procedure of Image Processing According to Second Embodiment]

Next, an image processing procedure according to the second embodimentwill be described with reference to FIG. 11. FIG. 11 is a flowchartillustrating a flow of a process according to the second embodiment ofthe present disclosure.

As illustrated in FIG. 11, the image processing apparatus 200 acquiresinformation regarding a user's motion from the HMD 10 as well asacquiring controller operation information from the controller 20 (stepS201). The processing of steps S202 to 204 is similar to steps S102 to104 illustrated in FIG. 6 and thus, description will be omitted.Following step S204, the image processing apparatus 200 determines thedisplay position of the virtual object in the processing area based onthe controller operation information (step S205). Subsequently, theimage processing apparatus 200 determines whether or not the motionspeed of the user is a predetermined threshold or less, similar to stepS105 illustrated in FIG. 6 (step S206).

In step S206, when the motion speed of the user is a predeterminedthreshold or less (step S206; Yes), the image processing apparatus 200further determines whether the motion speed of the virtual object is apredetermined threshold or less (step S207). In a case where the motionspeed of the virtual object is a predetermined threshold or less (stepS207; Yes), the image processing apparatus 200 analyzes the pixelinformation in the processing area including the virtual object (stepS208).

In contrast, in a case where the motion speed of the virtual objectexceeds the predetermined threshold (step S207; No), the imageprocessing apparatus 200 analyzes the pixel information in theprocessing area without including the virtual object (step S209).

Subsequently, the image processing apparatus 200 determines a variableused for tone mapping based on the analysis result of individual pixelsin the processing area (step S210). In a case where the user's motionspeed exceeds the predetermined threshold in step S206 (step S206; No),the image processing apparatus 200 reads out the already determinedreference luminance and variable (step S211).

Thereafter, the image processing apparatus 200 draws the virtual objectso as to be superimposed on the image of the processing area based onthe three-dimensional position of the virtual object or the like (stepS212). Furthermore, the image processing apparatus 200 converts (stepS213) the image of the processing area from HDR to SDR based on thereference luminance and the variables determined in step S210 or stepS211. Subsequently, the image processing apparatus 200 transmits theconverted image to the HMD 10 (step S214).

In this manner, the image processing apparatus 200 according to thesecond embodiment performs image processing according to the presentdisclosure on an image on which a virtual object is superimposed, basedon the distribution of individual pixels included in the image and thedistribution of individual pixels forming the virtual object. With thisconfiguration, the image processing apparatus 200 according to thesecond embodiment can perform more realistic and natural imageconversion that reflects changes in the brightness/darkness expressionwhen the sun light is blocked by hand.

4. Third Embodiment

[4-1. Configuration of Image Processing System According to ThirdEmbodiment]

Next, a third embodiment will be described. In the third embodiment, theimage processing according to the present disclosure is performed on thedisplay device side such as the HMD 30, rather than on the server sidesuch as the image processing apparatus 100.

An image processing system 3 according to the third embodiment will bedescribed with reference to FIG. 12. FIG. 12 is a diagram illustratingan example of the image processing system 3 according to the thirdembodiment of the present disclosure. As illustrated in FIG. 12, theimage processing system 3 includes: an HMD 30 including individualprocessing units included in the image processing apparatus 100according to the first embodiment and the image processing apparatus 200according to the second embodiment; and a storage server 300 having acontent storage unit 321.

The HMD 30 is an information device that includes a display 18A thatdisplays various types of content such as movies and still images andthat performs a conversion process on an SDR image displayed on thedisplay 18A. For example, the HMD 30 may be a smartphone VR goggle thatis realized by inserting a smartphone or the like into a goggle-shapedhousing.

An acquisition unit 31 corresponds to the detector 15 and theacquisition unit 131 described in the first embodiment and acquiresinformation regarding the user's motion on the HMD 30. An areadetermination unit 32 corresponds to the area determination unit 132described in the first embodiment. An analyzing unit 33 corresponds tothe analyzing unit 133 described in the first embodiment. The analyzingunit 33 acquires the spherical content stored in the storage server 300and analyzes the information of individual pixels in the processing areadetermined by the area determination unit 32, out of the sphericalcontent.

A variable determination unit 34 corresponds to the variabledetermination unit 134 described in the first embodiment. A converter 35corresponds to the converter 135 described in the first embodiment. Adisplay control unit 36 controls the process of displaying the imageconverted by the converter 35 on the display 18A.

In this manner, the HMD 30 according to the third embodiment functionsas an image processing apparatus that executes the image processingaccording to the present disclosure. That is, the HMD 30 can execute theimage processing according to the present disclosure as a standalonedevice, without depending on the server device or the like. Furthermore,the HMD 30 makes it possible to realize, as standalone operation, theprocesses including the display control process such as displaying theimage converted into SDR by the image processing according to thepresent disclosure on the display 18A. With this system, the HMD 30according to the third embodiment can implement the image processingaccording to the present disclosure with a simple system configuration.In addition, while the third embodiment is an example in which thespherical content is held in the storage server 300, the HMD 30 may holdthe spherical content in its own device.

5. Other Embodiments

The process according to each of embodiments described above may beperformed in various different forms (modifications) in addition to eachof embodiments described above.

For example, in each of the above-described embodiments, sphericalcontent is illustrated as an example of content. However, the imageprocessing according to the present disclosure can be applied to thecontent other than spherical content. For example, the image processingaccording to the present disclosure can be applied to a panoramic imageor a panoramic movie having a wider area than the displayable area ofthe display device. Moreover, application is also possible to VR imagesand VR movies formed in the range of 180 degrees. The spherical contentis not limited to still images and movies but may be game contentcreated in computer graphics (CG), for example.

Furthermore, each of the above-described embodiments is an example inwhich the first video data recorded in HDR is converted into the secondvideo data displayed in SDR. However, the image processing according tothe present disclosure is not limited to the process of converting HDRinto SDR. For example, the image processing according to the presentdisclosure can be applied to various types of processes for convertingan image recorded with a relatively high dynamic range into a relativelylow dynamic range.

Furthermore, each of the above-described embodiments is an example inwhich the image processing according to the present disclosure isexecuted based on the information of individual pixels included in theprocessing area corresponding to the image after the perspectiveprojection conversion of the display area. However, the image processingaccording to the present disclosure may be executed by using not onlythe information of individual pixels included in the processing area butalso the information of individual pixels of the area including a widerange of several pixels or several tens of pixels from the processingarea.

Furthermore, the image processing according to the present disclosuremay perform the conversion process not only on the image correspondingto the processing area but also on the image corresponding to the areaincluding a wide range of several pixels or tens of pixels from theprocessing area, for example. That is, the predetermined area as animage processing target does not necessarily indicate the area of theoriginal image after the projection conversion but may include the areaobtained by expanding the area corresponding to the image after theprojection conversion, by some pixels.

Furthermore, each of the above-described embodiments is an example inwhich tone mapping from HDR to SDR is performed based on Formulas (1) to(3) or the like. However, the conversion function of tone mapping forconverting the luminance value from HDR to SDR is not limited toFormulas (1) to (3). For example, various functions such as a well-knownlogarithmic function, power function, and Hill function used for tonemapping may be used as the conversion function of tone mapping. In anyof the functions, the image processing according to the presentdisclosure is realized by determining the reference luminance based onindividual pixels included in the processing area after projectionconversion or by determining the key value based on the distribution ofindividual pixels included in the processing area.

Furthermore, each of the above-described embodiments is the example inwhich the key value a is calculated based on Formula (4) using the modeof the luminance values of individual pixels included in the processingarea. However, the key value a does not necessarily have to becalculated using the mode. For example, the key value a may becalculated using various statistically significant numerical values suchas the median, the mean, and the standard deviation of the luminancevalues of the pixels included in the processing area.

Furthermore, the image processing according to the present disclosurehas been described as the process in which the display area is specifiedbased on the information regarding the motion of the user wearing theHMD 10 or the like (information regarding the inclination of the headposture or the line-of-sight direction). However, the informationregarding the user's motion is not limited to the above. For example, inthe case of displaying spherical content on a smartphone, tabletterminal, or the like, the user selects a display area by touchoperation on the screen or using an input device (mouse, trackpad, orthe like) in some cases. In this case, the information regarding theuser's motion includes information corresponding to the touch operationand information input via the input device. Furthermore, the motionspeed of the user includes information such as the speed of movement ofthe finger corresponding to the touch operation (in other words, themoving speed of a pointer in the tablet terminal), the moving speed ofthe pointer via the input device, or the like. In addition, theinformation regarding the motion of the user includes informationdetected by a sensor included in the tablet terminal when the user movesor tilts the tablet terminal. Furthermore, the information detected bythe sensor may include, for example, information such as the scrollingspeed of the screen (in other words, the processing area) on the tabletterminal.

Furthermore, among each process described in the above embodiments, allor a part of the processes described as being performed automaticallymay be manually performed, or the processes described as being performedmanually can be performed automatically by the known method. Inaddition, the processing procedures, specific names, and informationincluding various data and parameters illustrated in the above documentsor drawings can be arbitrarily changed unless otherwise specified. Forexample, various information illustrated in each of drawings is notlimited to the information illustrated.

In addition, each of components of each of devices is provided as afunctional and conceptional illustration and thus does not necessarilyneed to be physically configured as illustrated. That is, the specificform of distribution/integration of each of devices is not limited tothose illustrated in the drawings, and all or a part thereof may befunctionally or physically distributed or integrated into arbitraryunits according to various loads and use conditions. For example, theanalyzing unit 133 and the variable determination unit 134 illustratedin FIG. 1 may be integrated.

Furthermore, the above-described embodiments and modifications can beappropriately combined within a range implementable withoutcontradiction of processes.

The effects described in the present specification are merely examples,and thus, there may be other effects, not limited to the exemplifiedeffects.

6. Hardware Configuration

Information devices such as the image processing apparatus, HMD, andcontroller according to each of the embodiments described above areimplemented by a computer 1000 having the configuration as illustratedin FIG. 13, for example. Hereinafter, the image processing apparatus 100according to the first embodiment will be described as an example. FIG.13 is a hardware configuration diagram illustrating an example of thecomputer 1000 that implements the functions of the image processingapparatus 100. The computer 1000 includes a CPU 1100, RAM 1200, readonly memory (ROM) 1300, a hard disk drive (HDD) 1400, a communicationinterface 1500, and an input/output interface 1600. Each of componentsof the computer 1000 is interconnected by a bus 1050.

The CPU 1100 operates based on a program stored in the ROM 1300 or theHDD 1400 so as to control each of components. For example, the CPU 1100develops a program stored in the ROM 1300 or the HDD 1400 into the RAM1200 and executes processes corresponding to various programs.

The ROM 1300 stores a boot program such as a basic input output system(BIOS) executed by the CPU 1100 when the computer 1000 starts up, aprogram dependent on hardware of the computer 1000, or the like.

The HDD 1400 is a non-transitory computer-readable recording medium thatrecords a program executed by the CPU 1100, data used by the program, orthe like. Specifically, the HDD 1400 is a recording medium that recordsan image processing program according to the present disclosure, whichis an example of program data 1450.

The communication interface 1500 is an interface for connecting thecomputer 1000 to an external network 1550 (for example, the Internet).For example, the CPU 1100 receives data from other devices or transmitsdata generated by the CPU 1100 to other devices via the communicationinterface 1500.

The input/output interface 1600 is an interface for connecting betweenan input/output device 1650 and the computer 1000. For example, the CPU1100 receives data from an input device such as a keyboard or a mousevia the input/output interface 1600. In addition, the CPU 1100 transmitsdata to an output device such as a display, a speaker, or a printer viathe input/output interface 1600. Furthermore, the input/output interface1600 may function as a media interface for reading a program or the likerecorded on predetermined recording media. Examples of the media includeoptical recording media such as a digital versatile disc (DVD) or aphase change rewritable disk (PD), a magneto-optical recording mediumsuch as a magneto-optical disk (MO), a tape medium, a magnetic recordingmedium, and semiconductor memory.

For example, when the computer 1000 functions as the image processingapparatus 100 according to the first embodiment, the CPU 1100 of thecomputer 1000 executes the image processing program loaded on the RAM1200 to implement the functions of the acquisition unit 131 or the like.Furthermore, the HDD 1400 stores the image processing program accordingto the present disclosure or data in the content storage unit 121. Whilethe CPU 1100 executes the program data 1450 read from the HDD 1400, theCPU 1100 may acquire these programs from another device via the externalnetwork 1550, as another example.

Note that the present technology can also have the followingconfigurations.

(1)

An image processing apparatus comprising:

a variable determination unit that performs, based on a distribution ofluminance values of individual pixels included in a predetermined areadetermined according to information regarding a motion of a user out offirst video data recorded in a first dynamic range, determination of avariable to be used to calculate the luminance values of the individualpixels when the first dynamic range is converted into a second dynamicrange; and

a converter that converts the first video data into second video datadisplayed in the second dynamic range, based on the variable determinedby the variable determination unit.

(2)

The image processing apparatus according to (1),

wherein, in a case where the information regarding the motion of theuser satisfies a predetermined condition, the variable determinationunit determines the variable based on the distribution of the luminancevalues of the individual pixels included in the predetermined area.

(3)

The image processing apparatus according to (2),

wherein, in a case where a motion speed of the user is a predeterminedthreshold or less as a case where the information regarding the motionof the user satisfies the predetermined condition, the variabledetermination unit determines the variable based on the distribution ofthe luminance values of individual pixels included in the predeterminedarea.

(4)

The image processing apparatus according to (2) or (3),

wherein, in a case where the information regarding the motion of theuser does not satisfy the predetermined condition, the converterconverts the first video data into the second video data based on anexisting variable determined before a time point of determination of thepredetermined condition.

(5)

The image processing apparatus according to any one of (1) to (4),

wherein, in a case where pixel information of individual pixels includedin the predetermined area satisfies a predetermined condition, thevariable determination unit determines the variable based on adistribution of the luminance values of individual pixels included inthe predetermined area.

(6)

The image processing apparatus according to (5),

wherein, in a case where an amount of change of the pixel information ofindividual pixels included in the predetermined area in a predeterminedtime is a predetermined threshold or less as a case where the pixelinformation of individual pixels included in the predetermined areasatisfies the predetermined condition, the variable determination unitdetermines the variable based on the distribution of the luminancevalues of individual pixels included in the predetermined area.

(7)

The image processing apparatus according to (5) or (6),

wherein, in a case where the pixel information of the individual pixelsincluded in the predetermined area does not satisfy the predeterminedcondition, the converter converts the first video data into the secondvideo data based on an existing variable determined before a time pointof determination of the predetermined condition.

(8)

The image processing apparatus according to any one of (1) to (7),

wherein, in a case where the variable determination unit has determineda second variable after determining a first variable, the variabledetermination unit determines values that achieve stepwise transitionfrom the first variable to the second variable, and

the converter converts the first video data into the second video databy sequentially using the values that achieves stepwise transition fromthe first variable to the second variable, determined by the variabledetermination unit.

(9)

The image processing apparatus according to any one of (1) to (8),

wherein the variable determination unit determines the variable based ona value obtained by dividing a logarithmic mean of the luminance valuesof individual pixels included in the predetermined area by a mode of theluminance values of the individual pixels included in the predeterminedarea.

(10)

The image processing apparatus according to any one of (1) to (9),

wherein the variable determination unit determines the variable based ona distribution of luminance values of individual pixels included in avirtual object that is superimposed on the predetermined area andindividual pixels that are not superimposed on the virtual object in thepredetermined area.

(11)

The image processing apparatus according to (10),

wherein the variable determination unit calculates luminance values tobe set for individual pixels forming the virtual object, based onluminance values of individual pixels included in the first video data.

(12)

The image processing apparatus according to (10) or (11),

wherein in a case where a behavior of the virtual object satisfies apredetermined condition, the variable determination unit determines thevariable based on a distribution of luminance values of individualpixels included in a virtual object that is superimposed on thepredetermined area and individual pixels that are not superimposed onthe virtual object in the predetermined area, and

in a case where the behavior of the virtual object does not satisfy apredetermined condition, the variable determination unit determines thevariable based on the distribution of the luminance values of individualpixels included in the predetermined area regardless of whether thesuperimposition of the virtual object is performed.

(13)

The image processing apparatus according to any one of (1) to (12),

wherein the first video data is spherical content, and the imageprocessing apparatus further comprises

an area determination unit that performs projection conversion on apartial area of the spherical content specified based on the motion ofthe user and determines the area corresponding to the image after theprojection conversion, as the predetermined area.

(14)

The image processing apparatus according to any one of (1) to (13),further comprising

a display control unit that controls display of the second video dataobtained by conversion by the converter.

(15)

An image processing method, by a computer, comprising:

performing, based on a distribution of luminance values of individualpixels included in a predetermined area determined according toinformation regarding a motion of a user out of first video datarecorded in a first dynamic range, determination of a variable to beused to calculate the luminance values of the individual pixels when thefirst dynamic range is converted into a second dynamic range; and

converting the first video data into second video data displayed in thesecond dynamic range, based on the determined variable.

(16)

A non-transitory computer readable recording medium recording an imageprocessing program for causing a computer to function as:

a variable determination unit that performs, based on a distribution ofluminance values of individual pixels included in a predetermined areadetermined according to information regarding a motion of a user out offirst video data recorded in a first dynamic range, determination of avariable to be used to calculate the luminance values of the individualpixels when the first dynamic range is converted into a second dynamicrange; and

a converter that converts the first video data into second video datadisplayed in the second dynamic range, based on the variable determinedby the variable determination unit.

REFERENCE SIGNS LIST

-   -   1, 2, 3 IMAGE PROCESSING SYSTEM    -   10, 30 HMD    -   20 CONTROLLER    -   100, 200 IMAGE PROCESSING APPARATUS    -   300 STORAGE SERVER    -   15, 21 DETECTOR    -   16 TRANSMITTING UNIT    -   17 RECEIVING UNIT    -   18, 36 DISPLAY CONTROL UNIT    -   121, 321 CONTENT STORAGE UNIT    -   31, 131 ACQUISITION UNIT    -   32, 132 AREA DETERMINATION UNIT    -   33, 133, 233 ANALYZING UNIT    -   34, 134, 234 VARIABLE DETERMINATION UNIT    -   35, 135 CONVERTER    -   136 TRANSMITTING UNIT    -   235 DRAWING UNIT

1. An image processing apparatus comprising: a variable determinationunit that performs, based on a distribution of luminance values ofindividual pixels included in a predetermined area determined accordingto information regarding a motion of a user out of first video datarecorded in a first dynamic range, determination of a variable to beused to calculate the luminance values of the individual pixels when thefirst dynamic range is converted into a second dynamic range; and aconverter that converts the first video data into second video datadisplayed in the second dynamic range, based on the variable determinedby the variable determination unit.
 2. The image processing apparatusaccording to claim 1, wherein, in a case where the information regardingthe motion of the user satisfies a predetermined condition, the variabledetermination unit determines the variable based on the distribution ofthe luminance values of the individual pixels included in thepredetermined area.
 3. The image processing apparatus according to claim2, wherein, in a case where a motion speed of the user is apredetermined threshold or less as a case where the informationregarding the motion of the user satisfies the predetermined condition,the variable determination unit determines the variable based on thedistribution of the luminance values of individual pixels included inthe predetermined area.
 4. The image processing apparatus according toclaim 2, wherein, in a case where the information regarding the motionof the user does not satisfy the predetermined condition, the converterconverts the first video data into the second video data based on anexisting variable determined before a time point of determination of thepredetermined condition.
 5. The image processing apparatus according toclaim 1, wherein, in a case where pixel information of individual pixelsincluded in the predetermined area satisfies a predetermined condition,the variable determination unit determines the variable based on adistribution of the luminance values of individual pixels included inthe predetermined area.
 6. The image processing apparatus according toclaim 5, wherein, in a case where an amount of change of the pixelinformation of individual pixels included in the predetermined area in apredetermined time is a predetermined threshold or less as a case wherethe pixel information of individual pixels included in the predeterminedarea satisfies the predetermined condition, the variable determinationunit determines the variable based on the distribution of the luminancevalues of individual pixels included in the predetermined area.
 7. Theimage processing apparatus according to claim 5, wherein, in a casewhere the pixel information of the individual pixels included in thepredetermined area does not satisfy the predetermined condition, theconverter converts the first video data into the second video data basedon an existing variable determined before a time point of determinationof the predetermined condition.
 8. The image processing apparatusaccording to claim 1, wherein, in a case where the variabledetermination unit has determined a second variable after determining afirst variable, the variable determination unit determines values thatachieve stepwise transition from the first variable to the secondvariable, and the converter converts the first video data into thesecond video data by sequentially using the values that achievesstepwise transition from the first variable to the second variable,determined by the variable determination unit.
 9. The image processingapparatus according to claim 1, wherein the variable determination unitdetermines the variable based on a value obtained by dividing alogarithmic mean of the luminance values of individual pixels includedin the predetermined area by a mode of the luminance values of theindividual pixels included in the predetermined area.
 10. The imageprocessing apparatus according to claim 1, wherein the variabledetermination unit determines the variable based on a distribution ofluminance values of individual pixels included in a virtual object thatis superimposed on the predetermined area and individual pixels that arenot superimposed on the virtual object in the predetermined area. 11.The image processing apparatus according to claim 10, wherein thevariable determination unit calculates luminance values to be set forindividual pixels forming the virtual object, based on luminance valuesof individual pixels included in the first video data.
 12. The imageprocessing apparatus according to claim 10, wherein in a case where abehavior of the virtual object satisfies a predetermined condition, thevariable determination unit determines the variable based on adistribution of luminance values of individual pixels included in avirtual object that is superimposed on the predetermined area andindividual pixels that are not superimposed on the virtual object in thepredetermined area, and in a case where the behavior of the virtualobject does not satisfy a predetermined condition, the variabledetermination unit determines the variable based on the distribution ofthe luminance values of individual pixels included in the predeterminedarea regardless of whether the superimposition of the virtual object isperformed.
 13. The image processing apparatus according to claim 1,wherein the first video data is spherical content, and the imageprocessing apparatus further comprises an area determination unit thatperforms projection conversion on a partial area of the sphericalcontent specified based on the motion of the user and determines thearea corresponding to the image after the projection conversion, as thepredetermined area.
 14. The image processing apparatus according toclaim 1, further comprising a display control unit that controls displayof the second video data obtained by conversion by the converter.
 15. Animage processing method, by a computer, comprising: performing, based ona distribution of luminance values of individual pixels included in apredetermined area determined according to information regarding amotion of a user out of first video data recorded in a first dynamicrange, determination of a variable to be used to calculate the luminancevalues of the individual pixels when the first dynamic range isconverted into a second dynamic range; and converting the first videodata into second video data displayed in the second dynamic range, basedon the determined variable.
 16. A non-transitory computer readablerecording medium recording an image processing program for causing acomputer to function as: a variable determination unit that performs,based on a distribution of luminance values of individual pixelsincluded in a predetermined area determined according to informationregarding a motion of a user out of first video data recorded in a firstdynamic range, determination of a variable to be used to calculate theluminance values of the individual pixels when the first dynamic rangeis converted into a second dynamic range; and a converter that convertsthe first video data into second video data displayed in the seconddynamic range, based on the variable determined by the variabledetermination unit.